More operators for Databricks Repos #22422
Conversation
|
Can this functionality be done without adding a new operator? I am afraid of adding more operators when we do not even have plans on how we want to maintain them and test if they work properly. Commiets check that the change looks good. Sometimes they will look at the documentation and see if it matches what is implemeents, but it may not work properly. What can we do to improve the operational readiness of this provider, since we have recently had more changes in it? For Snowflake, we decided that we only have one operator, so it's easy to check that it's working properly. What would we like to do for Databricks? Do you have any plans? |
|
@mik-laj These operators are for specific functionality that is available via specific REST API, so alternative will be to get down to low-level API calls. It's a different from the Snowflake connector where you do everything via SQL - for Databricks we have similar We're planning to build a system test suite that will be used to test all operators. |
alexott
force-pushed
the
databricks-other-repos-operators
branch
from
a5d8d41 to
1fca768
Compare
Yep. We are just about to merge first change from AIP-47 #22311 and we will literally prepare (automatically) issues to convert all the providers. I will ask everyone invlved to comment and will assign them to those issues. |
| if repo_path is not None and not self.__repos_path_regexp__.match(repo_path): | ||
| raise AirflowException( | ||
| f"repo_path should have form of /Repos/{{folder}}/{{repo-name}}, got '{repo_path}'" | ||
| ) |
There was a problem hiding this comment.
Since repo_path is a templated field, there is a chance that the arg passed in is a Jinja expression or an XComArg in which case the regexp check would fail with a false-negative result potentially because templated values aren't evaluated until the task begins to execute.
Can you move this to the execute() method scope?
There was a problem hiding this comment.
thank you - I'll fix it later today.
There was a problem hiding this comment.
@potiuk @josh-fell I've addressed review comments...
alexott
force-pushed
the
databricks-other-repos-operators
branch
from
c953678 to
5e07b20
Compare
alexott
force-pushed
the
databricks-other-repos-operators
branch
from
e2879c9 to
109e3e3
Compare
ephraimbuddy
added
the
changelog:skip
This PR adds two additional operators for Databricks Repos:
DatabricksReposCreateOperator- to create a new Databricks RepoDatabricksReposDeleteOperator- to delete a Databricks Repo